

<!DOCTYPE html>

<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Benchmarking tips &#8212; LLVM 9 documentation</title>
    <link rel="stylesheet" href="_static/llvm-theme.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
    <script src="_static/jquery.js"></script>
    <script src="_static/underscore.js"></script>
    <script src="_static/doctools.js"></script>
    <script src="_static/language_data.js"></script>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="A guide to Dockerfiles for building LLVM" href="Docker.html" />
    <link rel="prev" title="Reporting Guide" href="ReportingGuide.html" />
<style type="text/css">
  table.right { float: right; margin-left: 20px; }
  table.right td { border: 1px solid #ccc; }
</style>

  </head><body>
<div class="logo">
  <a href="index.html">
    <img src="_static/logo.png"
         alt="LLVM Logo" width="250" height="88"/></a>
</div>

    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="Docker.html" title="A guide to Dockerfiles for building LLVM"
             accesskey="N">next</a> |</li>
        <li class="right" >
          <a href="ReportingGuide.html" title="Reporting Guide"
             accesskey="P">previous</a> |</li>
  <li><a href="http://llvm.org/">LLVM Home</a>&nbsp;|&nbsp;</li>
  <li><a href="index.html">Documentation</a>&raquo;</li>

        <li class="nav-item nav-item-this"><a href="">Benchmarking tips</a></li> 
      </ul>
    </div>


    <div class="document">
      <div class="documentwrapper">
          <div class="body" role="main">
            
  <div class="section" id="benchmarking-tips">
<h1>Benchmarking tips<a class="headerlink" href="#benchmarking-tips" title="Permalink to this headline">¶</a></h1>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
<p>For benchmarking a patch we want to reduce all possible sources of
noise as much as possible. How to do that is very OS dependent.</p>
<p>Note that low noise is required, but not sufficient. It does not
exclude measurement bias. See
<a class="reference external" href="https://www.cis.upenn.edu/~cis501/papers/producing-wrong-data.pdf">https://www.cis.upenn.edu/~cis501/papers/producing-wrong-data.pdf</a> for
example.</p>
</div>
<div class="section" id="general">
<h2>General<a class="headerlink" href="#general" title="Permalink to this headline">¶</a></h2>
<ul>
<li><p>Use a high resolution timer, e.g. perf under linux.</p></li>
<li><p>Run the benchmark multiple times to be able to recognize noise.</p></li>
<li><p>Disable as many processes or services as possible on the target system.</p></li>
<li><p>Disable frequency scaling, turbo boost and address space
randomization (see OS specific section).</p></li>
<li><p>Static link if the OS supports it. That avoids any variation that
might be introduced by loading dynamic libraries. This can be done
by passing <code class="docutils literal notranslate"><span class="pre">-DLLVM_BUILD_STATIC=ON</span></code> to cmake.</p></li>
<li><p>Try to avoid storage. On some systems you can use tmpfs. Putting the
program, inputs and outputs on tmpfs avoids touching a real storage
system, which can have a pretty big variability.</p>
<p>To mount it (on linux and freebsd at least):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">mount</span> <span class="o">-</span><span class="n">t</span> <span class="n">tmpfs</span> <span class="o">-</span><span class="n">o</span> <span class="n">size</span><span class="o">=&lt;</span><span class="n">XX</span><span class="o">&gt;</span><span class="n">g</span> <span class="n">none</span> <span class="n">dir_to_mount</span>
</pre></div>
</div>
</li>
</ul>
</div>
<div class="section" id="linux">
<h2>Linux<a class="headerlink" href="#linux" title="Permalink to this headline">¶</a></h2>
<ul>
<li><p>Disable address space randomization:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">echo</span> <span class="mi">0</span> <span class="o">&gt;</span> <span class="o">/</span><span class="n">proc</span><span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">kernel</span><span class="o">/</span><span class="n">randomize_va_space</span>
</pre></div>
</div>
</li>
<li><p>Set scaling_governor to performance:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">devices</span><span class="o">/</span><span class="n">system</span><span class="o">/</span><span class="n">cpu</span><span class="o">/</span><span class="n">cpu</span><span class="o">*/</span><span class="n">cpufreq</span><span class="o">/</span><span class="n">scaling_governor</span>
<span class="n">do</span>
  <span class="n">echo</span> <span class="n">performance</span> <span class="o">&gt;</span> <span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">devices</span><span class="o">/</span><span class="n">system</span><span class="o">/</span><span class="n">cpu</span><span class="o">/</span><span class="n">cpu</span><span class="o">*/</span><span class="n">cpufreq</span><span class="o">/</span><span class="n">scaling_governor</span>
<span class="n">done</span>
</pre></div>
</div>
</li>
<li><p>Use <a class="reference external" href="https://github.com/lpechacek/cpuset">https://github.com/lpechacek/cpuset</a> to reserve cpus for just the
program you are benchmarking. If using perf, leave at least 2 cores
so that perf runs in one and your program in another:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cset</span> <span class="n">shield</span> <span class="o">-</span><span class="n">c</span> <span class="n">N1</span><span class="p">,</span><span class="n">N2</span> <span class="o">-</span><span class="n">k</span> <span class="n">on</span>
</pre></div>
</div>
<p>This will move all threads out of N1 and N2. The <code class="docutils literal notranslate"><span class="pre">-k</span> <span class="pre">on</span></code> means
that even kernel threads are moved out.</p>
</li>
<li><p>Disable the SMT pair of the cpus you will use for the benchmark. The
pair of cpu N can be found in
<code class="docutils literal notranslate"><span class="pre">/sys/devices/system/cpu/cpuN/topology/thread_siblings_list</span></code> and
disabled with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">echo</span> <span class="mi">0</span> <span class="o">&gt;</span> <span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">devices</span><span class="o">/</span><span class="n">system</span><span class="o">/</span><span class="n">cpu</span><span class="o">/</span><span class="n">cpuX</span><span class="o">/</span><span class="n">online</span>
</pre></div>
</div>
</li>
<li><p>Run the program with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cset</span> <span class="n">shield</span> <span class="o">--</span><span class="n">exec</span> <span class="o">--</span> <span class="n">perf</span> <span class="n">stat</span> <span class="o">-</span><span class="n">r</span> <span class="mi">10</span> <span class="o">&lt;</span><span class="n">cmd</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>This will run the command after <code class="docutils literal notranslate"><span class="pre">--</span></code> in the isolated cpus. The
particular perf command runs the <code class="docutils literal notranslate"><span class="pre">&lt;cmd&gt;</span></code> 10 times and reports
statistics.</p>
</li>
</ul>
<p>With these in place you can expect perf variations of less than 0.1%.</p>
<div class="section" id="linux-intel">
<h3>Linux Intel<a class="headerlink" href="#linux-intel" title="Permalink to this headline">¶</a></h3>
<ul>
<li><p>Disable turbo mode:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">echo</span> <span class="mi">1</span> <span class="o">&gt;</span> <span class="o">/</span><span class="n">sys</span><span class="o">/</span><span class="n">devices</span><span class="o">/</span><span class="n">system</span><span class="o">/</span><span class="n">cpu</span><span class="o">/</span><span class="n">intel_pstate</span><span class="o">/</span><span class="n">no_turbo</span>
</pre></div>
</div>
</li>
</ul>
</div>
</div>
</div>


            <div class="clearer"></div>
          </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="Docker.html" title="A guide to Dockerfiles for building LLVM"
             >next</a> |</li>
        <li class="right" >
          <a href="ReportingGuide.html" title="Reporting Guide"
             >previous</a> |</li>
  <li><a href="http://llvm.org/">LLVM Home</a>&nbsp;|&nbsp;</li>
  <li><a href="index.html">Documentation</a>&raquo;</li>

        <li class="nav-item nav-item-this"><a href="">Benchmarking tips</a></li> 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
        &#169; Copyright 2003-2020, LLVM Project.
      Last updated on 2020-09-20.
      Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 3.2.1.
    </div>
  </body>
</html>